[Tune] Allow for Hierarchical Search Spaces in Ray1.0 #11216
Description
[Tune] @richardliaw
Both hyperopt and dragonfly documentation demonstrate applications with hierarchical search spaces. This sort of framework is beneficial for several reasons. Consistency of ray.tune between search spaces added as of Ray1.0 is a very nice feature, but search spaces such as:
space4svm = {
'C': hp.uniform('C', 0, 20),
'kernel': hp.choice('kernel', [
{'ktype': 'linear'},
{'ktype': 'poly', 'degree': hp.lognormal('degree', 0, 1)},
]),
'gamma': hp.uniform('gamma', 0, 20),
'scale': hp.choice('scale', [0, 1]),
'normalize': hp.choice('normalize', [0, 1])
}
HyperOptSearch(search_space=space4svm)
...are still unsupported. For example, when testing the hyperopt example provided at: https://docs.ray.io/en/releases-1.0.0/tune/examples/hyperopt_example.html, I added a the following config space as a toy example:
tune_kwargs = {
"num_samples": 4 ,
"config": {
"steps": 100,
"width": tune.uniform(0, 20),
"height": tune.uniform(-100, 100),
# This is an ignored parameter.
"activation": tune.choice(
[
None,
{
"type":"relu",
"percent_ranking": tune.uniform(0,10),
"fun": tune.choice(["foo","bar"])},
{
"type":"tanh",
"percent_ranking": tune.uniform(0,10),
"fun" : tune.choice(["lorem","ipsum"])
}
])
}}
when I run the same example (adapting functions to accommodate the new structure), ray does not properly cast the tune parameters inside the activation choices into e.g floats or strings. Instead, the percent ranking item is displayed as: <ray.tune.sample.Float object at 0x7fa4f02fdb70> and attempting to cast or add values yields something like: TypeError: unsupported operand type(s) for +: 'float' and 'Float'
It would be really nice to have some handling for this kind of space, even if it's limited to specific search algorithms.