Nice, this sounds like a nice feature for kaggle competitors :)

• About the isinstance issue, I don't think this is an issue. AFAIK there is no assumption that a ModelCV class should be a subclass of the vanilla Model class.
• Custom loss for model selection would be nice but can come in a later PR IMO
• Leveraging OOB samples for model selection and / or to return test-set accuracy estimate would be a boon too IMHO but can also probably be done in a later PR.
• Could you add some smoke tests for pickling and use in a pipeline / wrapping GridSearchCV (e.g. on the learning rate) so as to check that the base class g/setattr override does not break those meta tools? Maybe this can be done as part of the common testing framework by @amueller (I am not yet familiar with it).

@ogrisel I am not entirely sure what you want to check. Just that GridSearchCV works?
We could do that in the common test framework very easily for all classifiers. I am not sure if it is actually true, though. Well for multi-class it should be fine, binary might not work yet, see the inconsistent shape discussion.

indeed, maye a custom test for GBRT*CV models then. The params for grid is model specific anyway.

It may be nice to support tuning other parameters than n_estimators in GradientBoostingClassifierCV directly. Using GridSearchCV on top of GradientBoostingClassifierCV is possible but it would be semantically a bit different. It would result in a greedy approximation: tune n_estimators with the other parameters fixed first then tune the remaining of the parameters. On the other hand, if tuning other parameters were supported directly in GradientBoostingClassifierCV, it would be an exhaustive search. In term of implementation, the idea would be to generate parameter combinations with IterGrid but to handle n_estimators specifically for efficiency.

thanks for the feedback!

@mblondel I'm not a friend of supporting tuning other parameters; n_estimators has strong interactions with other tuning parameters (esp. learn_rate). E.g. changing learn_rate=0.001 to learn_rate=0.0001 likely requires 10x more boosting iterations for the same training deviance. In fact, I'd actually use GridSearchCV on top of GradientBoostingClassifierCV if I have sufficient computational resources.

If computational resources are an issue, I'd use GridSearchCV for paramter tuning choosing n_estimators as large as possible (depending on computational resources). Then, I'd use GradientBoostingClassifierCV to determine n_estimators.

@pprett Would be great to wrap up this in a "Parameters selection tips" section in the narrative doc of the GBRT models.

I agree but I would certainly end up copying Greg Ridgeways definitive guide to parameter selection in GBM (it is linked in the docs but it deserves more promotion) http://cran.r-project.org/web/packages/gbm/gbm.pdf .

@mblondel I'm not a friend of supporting tuning other parameters; n_estimators has strong interactions with other tuning parameters (esp. learn_rate). E.g. changing learn_rate=0.001 to learn_rate=0.0001 likely requires 10x more boosting iterations for the same training deviance. In fact, I'd actually use GridSearchCV on top of GradientBoostingClassifierCV if I have sufficient computational resources.

So, to summarize, you would use a sane default learning_rate and optimize n_estimators only in most cases?

If computational resources are an issue, I'd use GridSearchCV for paramter tuning choosing n_estimators as large as possible (depending on computational resources). Then, I'd use GradientBoostingClassifierCV to determine n_estimators.

I don't understand that. For me GradientBoostingClassifierCV should always be more efficient to choose n_estimators (the process is incremental and the prediction scores are readily available)

If computational resources are an issue, I'd use GridSearchCV for paramter tuning choosing n_estimators as large as possible (depending on computational resources). Then, I'd use GradientBoostingClassifierCV to determine n_estimators.

I don't understand that. For me GradientBoostingClassifierCV should always be more efficient to choose n_estimators (the process is incremental and the prediction scores are readily available)

Sorry, I meant I'd use GridSearchCV for tuning max_depth, min_samples_split and learn_rate using a fixed n_estimators (as large as possible, e.g. 3000). Then, I'd tune n_estimators via GradientBoostingClassifierCV fixing the other parameters with the values found by GridSearchCV. Does this makes sense now?

@pprett yes :)

@mblondel please ignore my first response to your comment - I was thinking about this issue yesterday and it does make sense to wrap GridSearchCV within GradientBoostingClassifierCV** - the way I described it here would be rather wasteful in terms of computational resources (2x CV)... Apart from that I'm not totally sure about the difference between the two approaches... I need to spend more time on this issue. Anyway, sorry for the noise.

** also described in this thread on the ML http://www.mail-archive.com/scikit-learn-general@lists.sourceforge.net/msg03395.html

PS: I promise next time I'll think before I write

No worries.

To tune other parameters than n_estimators directly in GradientBoostingClassifierCV, the idea that I had in mind was to use IterGrid to generate all parameter combinations (n_estimators excepted). For example, given {'learn_rate':[0.05, 0.01, 0.001], 'subsample':[0.25, 0.5, 0.75]}, one parameter combination will be {'learn_rate'=0.05, 'subsample'=0.25}. Given these two values fixed, it is possible to choose n_estimators efficiently. Then this process must be repeated for each fold (to compute the average score of a parameter combination).

One thing that worries me about using GridSearchCV on top of GradientBoostingClassifierCV is that the train / validation split will have to be made twice (once inside GridSearchCV, and once inside GradientBoostingClassifierCV). Not good if you don't have so much data.

Supporting other parameters in GradientBoostingClassifierCV will however increase the implementation complexity...

2012/8/22 Mathieu Blondel notifications@github.com

No worries.

To tune other parameters than n_estimators directly in
GradientBoostingClassifierCV, the idea that I had in mind was to use
IterGrid to generate all parameter combinations (n_estimators excepted).
For example, given {'learn_rate':[0.05, 0.01, 0.001], 'subsample':[0.25,
0.5, 0.75]}, one parameter combination will be {'learn_rate'=0.05,
'subsample'=0.25}. Given these two values fixed, it is possible to choose
n_estimators efficiently. Then this process must be repeated for each
fold (to compute the average score of a parameter combination).

Exactly, for each grid point and fold you get an array of deviance scores
(shape=n_estimators); for each grid point you then need to compute the mean
deviance scores across all folds and pick n_estimators with the lowest
(mean) deviance.

One thing that worries me about using GridSearchCV on top of
GradientBoostingClassifierCV is that the train / validation split will
have to be made twice (once inside GridSearchCV, and once inside
GradientBoostingClassifierCV). Not good if you don't have so much data.

I agree

Supporting other parameters in GradientBoostingClassifierCV will however
increase the implementation complexity...

IterGrid and KFold do most of the heavy lifting - I simply pass the
results of each grid point - fold combination into itertools.groupby in
order to groupby grid point id and than compute mean deviance for each grid
point. I'll push an update in the evening.

thanks!

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1036#issuecomment-7927499.

Peter Prettenhofer

On Wed, Aug 22, 2012 at 01:18:28AM -0700, Mathieu Blondel wrote:

One thing that worries me about using GridSearchCV on top of
GradientBoostingClassifierCV is that the train / validation split will
have to be made twice (once inside GridSearchCV, and once inside
GradientBoostingClassifierCV). Not good if you don't have so much data.

That's one reason why we need to be able to have cross-validation like
objects use a validation set. There is (quite a bit) of design work to do
here...

G

Did someone say api design?

If we where able to pass the CV-like object the test-split of the grid-search, we'd be fine, right?
So we add another function fit_with_validation(X_train, y, X_test) and let the GridSearchCV check if the object has that,
the splitting-twice problem would be averted.

Revived by @vighneshbirodkar in #5689

Fixed in #7071