Ask and tell interface tutorial by nzw0301 · Pull Request #2422 · optuna/optuna

nzw0301 · 2021-02-27T17:37:37Z

Motivation

Optuna has introduced an awesome interface called ask and tell. This pr aims to give a tutorial for the interface.

Description of the changes

Write the tutorial covering the following three features of ask-and-tell
- Apply optuna to an existing optimization problem with minimum modifications
- Define and Run
- Batch Optimization
Mention the tutorial in ask and tell methods. (Thanks @crcrpar for your suggestions!)

nzw0301 · 2021-02-27T17:48:44Z

By the way, does formats.sh check the tutorial? I needed to run black tutorial/ on my local machine...

codecov-io · 2021-02-27T18:00:23Z

Codecov Report

Merging #2422 (a28591a) into master (30f58b5) will decrease coverage by 0.07%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2422      +/-   ##
==========================================
- Coverage   91.44%   91.36%   -0.08%     
==========================================
  Files         134      135       +1     
  Lines       11270    11307      +37     
==========================================
+ Hits        10306    10331      +25     
- Misses        964      976      +12

Impacted Files	Coverage Δ
optuna/study.py	`96.13% <ø> (ø)`
optuna/integration/pytorch_lightning.py	`83.33% <0.00%> (-4.17%)`	⬇️
optuna/integration/allennlp.py	`88.11% <0.00%> (-1.00%)`	⬇️
optuna/importance/_fanova/_tree.py	`99.45% <0.00%> (-0.01%)`	⬇️
optuna/integration/botorch.py	`97.75% <0.00%> (ø)`
optuna/storages/_rdb/models.py	`99.59% <0.00%> (ø)`
optuna/samplers/_search_space.py	`100.00% <0.00%> (ø)`
optuna/storages/_cached_storage.py	`97.06% <0.00%> (ø)`
optuna/samplers/_tpe/parzen_estimator.py	`100.00% <0.00%> (ø)`
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30f58b5...a28591a. Read the comment docs.

crcrpar

Thank you for your great work.

I think it's better to refer to this tutorial from ask and tell methods.

What's your take?

nzw0301 · 2021-02-28T07:46:44Z

@crcrpar Absolutely!

keisuke-umezawa · 2021-02-28T09:02:50Z

@hvy could you also review it?

nzw0301 · 2021-02-28T10:01:47Z

I'll update the PR as suggested by @crcrpar in a few days; refer to the tutorial from ask and tell methods.

Cheers,

hvy · 2021-03-01T01:37:36Z

Just for the record, this is a tracking issue for ask and tell related changes #2177

crcrpar

Basically okay. I'm not quite happy with my suggestions thus comments and it's up to you and other reviewers.

crcrpar · 2021-03-01T08:27:11Z

+
+.. _Apply-optuna-to-an-existing-optimization-problem-with-minimum-modifications:
+
+----------------------------------------------------------------------------


nit-picking this line isn't necessary. But I like this because this line makes the structure clearer while editing.

Suggested change

----------------------------------------------------------------------------

----------------------------------------------------------------------------

crcrpar · 2021-03-05T11:12:57Z

+# A drawback of this traditional interface is less flexibility.
+# If ``objective`` requires an additional argument for ``objective``,
+# you need to define your customized class for the objective as in
+# `this FAQ section <../../faq.html#how-to-define-objective-functions-that-have-own-arguments>`_.
+# That will make the codebase more complicated.
+# The ask-and-tell interface provides a more flexible syntax to optimize hyperparameters.
+# The following example of ask-and-tell interface is equivalent to the previous code block.


I do prefer a shorter description, but let @Crissman decide.

The following is my suggestion and not a strong suggestion.

Suggested change

# A drawback of this traditional interface is less flexibility.

# If ``objective`` requires an additional argument for ``objective``,

# you need to define your customized class for the objective as in

# `this FAQ section <../../faq.html#how-to-define-objective-functions-that-have-own-arguments>`_.

# That will make the codebase more complicated.

# The ask-and-tell interface provides a more flexible syntax to optimize hyperparameters.

# The following example of ask-and-tell interface is equivalent to the previous code block.

# This interface is not flexible enough.

# For example, if ``objective`` requires additional arguments other than ``trial``,

# you need to define a class as in

# `How to define objective functions that have own arguments? <../../faq.html#how-to-define-objective-functions-that-have-own-arguments>`_.

# The ask-and-tell interface provides a more flexible syntax to optimize hyperparameters.

# The following example is equivalent to the previous code block.

Aside from changing that first line to explain why ask/tell is not always superior, I like @crcrpar 's shorter description. 🙇‍♂️

Sorry but I don't understand your comment enough. Do you mean it's better to leave the first line "A drawback of this traditional interface is less flexibility." as is and replace the other line with my suggestion?

Co-authored-by: Masaki Kozuki <masaki.kozuki.2014@gmail.com>

Crissman

Good tutorial. Thanks!

Crissman · 2021-03-10T00:59:46Z

+# A drawback of this traditional interface is less flexibility.
+# If ``objective`` requires an additional argument for ``objective``,
+# you need to define your customized class for the objective as in
+# `this FAQ section <../../faq.html#how-to-define-objective-functions-that-have-own-arguments>`_.
+# That will make the codebase more complicated.
+# The ask-and-tell interface provides a more flexible syntax to optimize hyperparameters.
+# The following example of ask-and-tell interface is equivalent to the previous code block.


Aside from changing that first line to explain why ask/tell is not always superior, I like @crcrpar 's shorter description. 🙇‍♂️

Crissman

The separate examples structure is looking good! Let's sell the tutorial more in the API doc.

Co-authored-by: Crissman Loomis <crissman@preferred.jp>

Crissman

LGTM!

nzw0301 · 2021-03-17T05:22:39Z

Thank you for your long reviews! I'm happy to learn this interface!

crcrpar reviewed Feb 28, 2021

View reviewed changes

keisuke-umezawa assigned hvy and crcrpar Feb 28, 2021

crcrpar added document Documentation related. sprint-20210227 PR from the online sprint event Feb 27, 2021. labels Mar 1, 2021

nzw0301 force-pushed the add_ask-and-tell_tutorial branch from 8c38403 to eeb6800 Compare March 1, 2021 05:53

nzw0301 added 8 commits March 1, 2021 14:56

Ask and tell interface tutorial

99d98bd

Apply black

53e02d2

Unify the termiology

55b1022

Specify the direction

852fdc0

Add sphinx.ext.autosectionlabel

f762a11

Add seealso for the tutorial page

c053459

Use autolabel syntax

0239e17

Improve the sentence of seealso

5340398

nzw0301 force-pushed the add_ask-and-tell_tutorial branch from eeb6800 to 5340398 Compare March 1, 2021 05:56

g-votte assigned Crissman and unassigned hvy Mar 1, 2021

nzw0301 added 2 commits March 1, 2021 17:18

Revert commits related to the extension

8a2e83d

Replace optuna with Optuna

0241425

crcrpar reviewed Mar 5, 2021

View reviewed changes

nzw0301 and others added 4 commits March 5, 2021 22:01

Update tutorial/20_recipes/009_ask_and_tell.py

cd33ee2

Co-authored-by: Masaki Kozuki <masaki.kozuki.2014@gmail.com>

Update tutorial/20_recipes/009_ask_and_tell.py

71df0e1

Co-authored-by: Masaki Kozuki <masaki.kozuki.2014@gmail.com>

Update tutorial/20_recipes/009_ask_and_tell.py

096e242

Co-authored-by: Masaki Kozuki <masaki.kozuki.2014@gmail.com>

Update tutorial/20_recipes/009_ask_and_tell.py

cd24d33

Co-authored-by: Masaki Kozuki <masaki.kozuki.2014@gmail.com>

Crissman suggested changes Mar 10, 2021

View reviewed changes

Update setences by following the review

a28591a

nzw0301 mentioned this pull request Mar 11, 2021

Document how to write a new tutorial in CONTRIBUTING.md #2463

Merged

Crissman suggested changes Mar 15, 2021

View reviewed changes

Comment thread optuna/study.py Outdated

Comment thread optuna/study.py Outdated

nzw0301 and others added 2 commits March 15, 2021 14:35

Update optuna/study.py

7f241e4

Co-authored-by: Crissman Loomis <crissman@preferred.jp>

Update optuna/study.py

ef4e921

Co-authored-by: Crissman Loomis <crissman@preferred.jp>

Crissman approved these changes Mar 16, 2021

View reviewed changes

crcrpar approved these changes Mar 16, 2021

View reviewed changes

Crissman merged commit 4891377 into optuna:master Mar 17, 2021

Crissman added this to the v2.6.1 milestone Mar 17, 2021

nzw0301 deleted the add_ask-and-tell_tutorial branch March 17, 2021 05:21

toshihikoyanase modified the milestones: v2.6.1, v2.7.0 Apr 2, 2021

hvy mentioned this pull request May 10, 2021

Batch optimization #2660

Closed


		.. _Apply-optuna-to-an-existing-optimization-problem-with-minimum-modifications:

		----------------------------------------------------------------------------

Uh oh!

Conversation

nzw0301 commented Feb 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description of the changes

Uh oh!

nzw0301 commented Feb 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-io commented Feb 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

crcrpar left a comment

Choose a reason for hiding this comment

Uh oh!

nzw0301 commented Feb 28, 2021

Uh oh!

keisuke-umezawa commented Feb 28, 2021

Uh oh!

nzw0301 commented Feb 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hvy commented Mar 1, 2021

Uh oh!

crcrpar left a comment

Choose a reason for hiding this comment

Uh oh!

crcrpar Mar 1, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

crcrpar Mar 5, 2021

Choose a reason for hiding this comment

Uh oh!

Crissman Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

crcrpar Mar 11, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Crissman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Crissman Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Crissman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Crissman left a comment

Choose a reason for hiding this comment

Uh oh!

nzw0301 commented Mar 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

nzw0301 commented Feb 27, 2021 •

edited

Loading

nzw0301 commented Feb 27, 2021 •

edited

Loading

codecov-io commented Feb 27, 2021 •

edited

Loading

nzw0301 commented Feb 28, 2021 •

edited

Loading