Refactoring of successive halving.#808
Merged
toshihikoyanase merged 3 commits intooptuna:masterfrom Dec 20, 2019
Merged
Conversation
Merged
sile
approved these changes
Dec 19, 2019
Member
There was a problem hiding this comment.
Thank you for your PR.
I executed a benchmarking to confirm that this PR does not cause unintended behavioral changes (see below).
// Install kurobako (benchmark tool).
$ curl -L https://github.com/sile/kurobako/releases/download/0.1.3/kurobako-0.1.3.linux-amd64 -o kurobako
$ chmod +x kurobako && sudo mv kurobako /usr/local/bin/
// Download benchmark data file.
$ curl -OL http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz
$ tar xf fcnet_tabular_benchmarks.tar.gz && cd fcnet_tabular_benchmarks/
// Create a problem recipe.
$ kurobako problem hpobench fcnet_parkinsons_telemonitoring_data.hdf5 > problem.json
// Benchmark `SuccessiveHalvingPruner` by using `optuna/optuna:0.19.0`.
$ pip uninstall -y optuna; pip install -U optuna==0.19.0
$ kurobako solver --name 'SuccessiveHalving@0.19.0' optuna --pruner asha > solver-0.19.0.json
$ kurobako studies --solvers $(cat solver-0.19.0.json) --problems $(cat problem.json) --repeats 1 --seed 1 | kurobako run > result-0.19.0.json
// Benchmark `SuccessiveHalvingPruner` by using `hvy/optuna:refactor-successive-halving`.
$ pip install git+https://github.com/hvy/optuna@refactor-successive-halving
$ kurobako solver --name 'SuccessiveHalving@refactor-successive-halving' optuna --pruner asha > solver-refactor-successive-halving.json
$ kurobako studies --solvers $(cat solver-refactor-successive-halving.json) --problems $(cat problem.json) --repeats 1 --seed 1 | kurobako run > result-refactor-successive-halving.json
// Compare the optimization (benchmarking) results.
$ jq '.trials[].evaluations[].values' result-0.19.0.json | head
[
0.4457722008228302
]
[
0.43835049867630005
]
[
0.4163028597831726
]
$ jq '.trials[].evaluations[].values' result-0.19.0.json | wc -l
5997
$ jq '.trials[].evaluations[].values' result-0.19.0.json | md5sum
63f3ce3900f7b230e51bc5b954b2fa45 -
$ jq '.trials[].evaluations[].values' result-refactor-successive-halving.json | md5sum
63f3ce3900f7b230e51bc5b954b2fa45 -As you can see the outputs of the last two commands, the SuccessiveHalvingPruner in this PR branch produced the exact same optimization result with the v0.19.0 one (i.e., there is no performance degradation).
Codecov Report
@@ Coverage Diff @@
## master #808 +/- ##
==========================================
+ Coverage 90.15% 90.15% +<.01%
==========================================
Files 106 106
Lines 8768 8769 +1
==========================================
+ Hits 7905 7906 +1
Misses 863 863
Continue to review full report at Codecov.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR can probably wait until #785 is merged. I was mostly looking through the code and made these changes for my own readability.A minor refactoring of the successive halving pruner that reduces calls to
_completed_rung_key, changes some variable named and introduces indirections.If you find the indirections to increase the complexity, please feel free to close this PR.