[MRG] Some more tests #44
Conversation
glouppe
commented
Apr 5, 2016
|
This adds some more tests. I'll merge if Travis is happy. |
| yield (check_minimize, bench2, -5, [[-6, 6]], 0.05, 10000) | ||
| yield (check_minimize, bench3, -0.9, [[-2, 2]], 0.05, 10000) | ||
| yield (check_minimize, branin, 0.39, [[-5, 10], [0, 15]], 0.1, 10000) | ||
| yield (check_minimize, hart6, -3.32, np.tile((0, 1), (6, 1)), 0.5, 10000) |
There was a problem hiding this comment.
Why bother making it complicated when this passes? :( @betatim @MechCoder
There was a problem hiding this comment.
Great question.
Well, these are "simple" functions. We should investigate complicated functions where we can gain some information from the data, for instance LogisticRegression, where we can choose hyperparameters close to those that have high regularization.
LogisticRegression might be a bad example because again we have to specify the scale in which we need to query the parameters.
|
Less iterations? I think that is the big selling point. Does it make sense to add a test (or should it be a benchmark?) that puts the "clever" algorithms in a race against Vaguely remember from the "Random Grid Search" Bergstra and company paper that they conclude that random search is pretty optimal (scientific term :) ) for ~high dimensions if some of the dimensions don't actually matter. You could only beat it if you knew which dimensions didn't matter and removed them. Something along those lines. Anyway, if we can't beat |
|
@betatim Yes, I think that should be the first step from now onwards before implementing new stuff.
|
|
Should we rename dummy -> random? In particular if it continues to be useful/competitive we might want to give it a less patronising name. |
[WIP] Eval PR #579 - Plotting and names