-
Notifications
You must be signed in to change notification settings - Fork 7.4k
Description
Describe your feature request
I'm working on a project where we want the user to be able to choose which algorithm they want to use via the command line. This makes for ugly code—for example, see:
https://github.com/MLTransfer/MLEncrypt-Research/blob/ce2c6b5fbf77d2486ec4a702a4bdf63dcdba2330/mlencrypt_research/cli.py#L446-L635
Dedicating almost 200 lines of code to instantiating the search algorithm seems unnatural, and this same problem occurs for choosing schedulers.
At least for search algorithms, this is mainly because most algos only work with 1 hparam configuration format. I think a reasonable way to deal with this is to force the user to stick to a single format, and then tune converts this to the appropriate format. Using ConfigSpace seems reasonable since it's meant for hparam configs and is already required by BOHB.
One problem with implementing this is that different search algorithms accept different arguments. An easy way to work around this is to just accept **kwargs in the shim and pass these to the initializer of the actual search algorithm object.
TL;DR something that looks like this:
hparam_config = ConfigSpace() # properly instantiate this
search_alg = ray.tune.suggest.create('dragonfly', hparam_config, **kwargs)
scheduler = ray.tune.schedulers.create('asha', **kwargs)
tune.run(trainable, search_alg=search_alg, scheduler=scheduler)Originally suggested to @richardliaw in his Slack DMs.
edit: see #10451 for shim instantiation