nan handling of optuna tell. #3132
Description
When we perform optuna tell with nan, you'll get the following error message.
$ optuna create-study --storage sqlite:///debug.db --study-name foo
[I 2021-11-29 15:22:23,775] A new study created in RDB with name: foo
foo
$ optuna ask --storage sqlite:///debug.db --study-name foo
.../python3.8/site-packages/optuna/cli.py:796: ExperimentalWarning: 'ask' is an experimental CLI command. The interface can change in the future.
warnings.warn(
[I 2021-11-29 15:22:31,214] Using an existing study with name 'foo' instead of creating a new one.
[I 2021-11-29 15:22:31,267] Asked trial 0 with parameters {}.
{"number": 0, "params": {}}
$ optuna tell --storage sqlite:///debug.db --study-name foo --trial-number 0 --values nan
.../python3.8/site-packages/optuna/cli.py:863: ExperimentalWarning: 'tell' is an experimental CLI command. The interface can change in the future.
warnings.warn(
[E 2021-11-29 15:22:43,074] Trial 0 failed, because the objective function returned nan.
However, the state of trial 0 remains RUNNING.
In [1]: import optuna
In [2]: optuna.load_study(storage="sqlite:///debug.db", study_name="foo").get_trials()[0]
Out[2]: FrozenTrial(number=0, values=None, datetime_start=datetime.datetime(2021, 11, 29, 15, 22, 31, 248437), datetime_complete=None, params={}, distributions={}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=1, state=TrialState.RUNNING, value=None)Expected behavior
I think there are several "correct" behavior. I personally prefer pattern A but this might not be optimal from a more general perspective.
pattern A
optuna tell (more precisely, study.tell) set the study's state to FAILED and raise no errors just like study.optimize.
pattern B
optuna tell (more precisely, study.tell) set the study's state to FAILED and raise an error.
pattern C
Change the error message.
Environment
- Optuna version: a3.0.0a1.dev
- Python version: 3.8.6
- OS: Linux-5.10.60.1-microsoft-standard-WSL2-x86_64-with-glibc2.29
- (Optional) Other libraries and their versions:
Additional context (optional)
None, null, and other nans:
$ optuna tell --storage sqlite:///debug.db --study-name foo --trial-number 0 --values NaN
/home/ytsmiling/.venv/lib/python3.8/site-packages/optuna/cli.py:863: ExperimentalWarning: 'tell' is an experimental CLI command. The interface can change in the future.
warnings.warn(
[E 2021-11-29 15:43:17,883] Trial 0 failed, because the objective function returned nan.
$ optuna tell --storage sqlite:///debug.db --study-name foo --trial-number 0 --values np.nan
usage: optuna tell [-h] [--study-name STUDY_NAME] [--trial-number TRIAL_NUMBER] [--values VALUES [VALUES ...]] [--state STATE]
optuna tell: error: argument --values: invalid float value: 'np.nan'
[E 2021-11-29 15:43:23,779]
$ optuna tell --storage sqlite:///debug.db --study-name foo --trial-number 0 --values None
usage: optuna tell [-h] [--study-name STUDY_NAME] [--trial-number TRIAL_NUMBER] [--values VALUES [VALUES ...]] [--state STATE]
optuna tell: error: argument --values: invalid float value: 'None'
[E 2021-11-29 15:33:50,293]
$ optuna tell --storage sqlite:///debug.db --study-name foo --trial-number 0 --values null
usage: optuna tell [-h] [--study-name STUDY_NAME] [--trial-number TRIAL_NUMBER] [--values VALUES [VALUES ...]] [--state STATE]
optuna tell: error: argument --values: invalid float value: 'null'
[E 2021-11-29 15:33:57,260]