This estimator does not support Dask dataframes

Question

I am trying to fit a model using Dask framework, and the estimator used in example says it does not accept Dask dataframe. Can someone help me please ?

    from dask_ml.model_selection import IncrementalSearchCV
    from sklearn.linear_model import SGDClassifier

    ddx,ddy = dd.from_pandas(X,chunksize=100000), 
    dd.from_pandas(y,chunksize=100000)
    X_train, X_test, y_train, y_test = train_test_split(ddx, ddy)
    model = SGDClassifier(loss='log')
    params = { 'alpha': np.logspace(-2, 1, num=1000) }
    search = IncrementalSearchCV(model, params,
                         n_initial_parameters=10, random_state=0)
    search.fit(X_train, y_train, classes=classes)
    y_pred = search.predict_proba(X_test)

The error log is : TypeError: This estimator does not support dask dataframes.

It appears on the search fit line. When I replace by partial_fit it works but then the same error happens on the predict_proba line.

Please provide more context. More specifically - how do you import the SGDClassifier and IncrementalSearchCV? Also, is this the row where the error occurs - search.fit(X_train, y_train, classes=classes)? — stan0
– stan0, Commented Mar 25, 2020 at 10:24
Also, everything works fine when i use Incremental instead of IncrementalSearchCV but the issue is : I need to optimize the hyperparameters. — Maxime Heuillet
– Maxime Heuillet, Commented Mar 25, 2020 at 11:09
Is there a chance dask_ml also provides an SGDClassifier? Maybe you should use the "dask version" if there is one, of course. — stan0
– stan0, Commented Mar 25, 2020 at 12:15
Also, if you update your question with what you told me in the comments (here), it would be better for others who read it and would be able to help you more :) — stan0
– stan0, Commented Mar 25, 2020 at 12:15
Dask uses these imports on its quick-run examples so it should work with these imports — Maxime Heuillet
– Maxime Heuillet, Commented Mar 25, 2020 at 12:34

mdurant · Accepted Answer · 2020-03-25 18:45:48Z

2

IncrementalSearchCV currently requires Dask Arrays, perhaps you can convert your data.

I opened https://github.com/dask/dask-ml/issues/628 to suport dataframes. Would welcome help if you're interested in working on it.

edited Mar 25, 2020 at 18:45

mdurant

28.9k5 gold badges49 silver badges79 bronze badges

answered Mar 25, 2020 at 14:42

TomAugspurger

29.1k8 gold badges90 silver badges71 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Maxime Heuillet Over a year ago

Hello, my X_train data has several columns. What you are saying is that I can't use IncrementalSearchCV to optimize the model for now ? How can I help on the Github issue ?

Collectives™ on Stack Overflow

This estimator does not support Dask dataframes

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related