Reference Issues/PRs

Addresses (but doesn't fix) #1626: "API Proposal: Genearlized Cross-Validation and Early Stopping". This PR only address early stopping on models that have partial_fit (#1626 (comment)), not addressing "when more computation doesn't improve the result." (#1626 (comment)).

I will use this in dask/dask-searchcv#72. Briefly, I want to repeatedly use cross_validate with a specified number of partial_fit calls.

What does this implement/fix? Explain your changes.

This PR implements early stopping for cross validation when partial_fit is present. This requires that

• _fit_and_score have a partial_fit keyword argument that can either be a bool or an int that reflects the number of times est.partial_fit is called.
  • I also add a partial_fit keyword arg to cross_validate, which relies on _fit_and_score.
• the specification of a different train/validation set if _fit_and_score is called repeatedly.
  • yes, it's possible to make sure train/validation are always separated on repeated calls but not really user-friendly and doesn't have the cleanest documentation.

TODO

• make work with pipelines
• allow for specification of a separate train and validation set as mentioned in API Proposal: Genearlized Cross-Validation and Early Stopping #1626 (comment)
• make sure will as expected when called repeatedly, which comes up in ENH: Hyperband implementation dask/dask-searchcv#72.
• integrate with GridSearchCV and RandomizedSearchCV