I know this is a restriction if pickle is used but is this still needed now that joblib is using cloudpickle?

I know this is a restriction if pickle is used but is this still needed now that joblib is using cloudpickle?

@albertcthomas would it make sense to add this for pre-cloudpickle versions of sklearn ? Could you let me know which version onwards has cloudpickle ?

I was just wondering. It might be relevant even with cloudpickle. cc @tomMoral

Thanks. Can you test if the issue persists in 0.20.1? If so, then cloudpickle doesn't help.

Installed 0.20.1 version of scikit-learn and this code seems to run fine without loading the custom scorer from an external module.

from sklearn.model_selection import cross_val_score, RepeatedKFold
from sklearn.linear_model import ElasticNet
from sklearn.preprocessing import StandardScaler
import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.metrics import make_scorer, mean_squared_error

def rmse_cv(y_true, y_pred) :
	assert len(y_true) == len(y_pred)
	y_pred = np.exp(y_pred)
	y_true = np.exp(y_true)
	return np.sqrt(mean_squared_error(y_true,y_pred))


rkfold = RepeatedKFold(n_splits=5,n_repeats=5)

url = 'https://github.com/GinoWoz1/Learnings/raw/master/'

X_train = pd.read_csv(url + 'X_trainGA.csv',index_col= 'Unnamed: 0')
y_train = pd.read_csv(url +'y_trainGA.csv',header=None,index_col=0)

X_train.rename(columns={'Constant Term':'tax'},inplace=True)

elnet_final = ElasticNet()

elnet_pipe = Pipeline([('std',StandardScaler()),
('elnet',elnet_final)])

cross_val = cross_val_score(elnet_pipe,
						X_train,
						y_train,
						scoring=make_scorer(rmse_cv, greater_is_better=False),
						cv=rkfold,
						n_jobs=-1)

So now that joblib is using cloudpickle I think that it should work out of the box for most of the cases (if the loky backend is used). Instead of the example we can maybe add a remark in the doc saying that if a custom scoring function is used with n_jobs >1 then it must be serializable by joblib and we can point to the joblib documentation section explaining how serialization is done

Thanks, @albertcthomas.

@amueller @adrinjalali - comments ?

@jnothman The new Loky backend takes care of serialising the interactively-defined functions, but I'm not sure if older versions of sklearn will work with this new backend. Can I suggest this to be kept for reasons of backward compatibility ?

Can I suggest this to be kept for reasons of backward compatibility ?

I think it would be better to say that this example is useful as other backends than loky (multiprocessing, custom backends) can be used.

Roger that, @albertcthomas.

Well it would be nice if someone could confirm that this is the expected behavior when loky is used and when a different backend is used

Regarding the rest: rmse is only a placeholder-name for a custom function which is
not available as a custom scorer via sklearn, directly.

IMO rmse would be fine if it was defined in the example and I agree that most of the users will know that rmse stands for root mean square error but better explicit than implicit.

Regarding the rest: rmse is only a placeholder-name for a custom function which is
not available as a custom scorer via sklearn, directly.

IMO rmse would be fine if it was defined in the example and I agree that most of the users will know that rmse stands for root mean square error but better explicit than implicit.

We could keep an abstract function name, like 'custom_scoring_function`, probably ?

We could keep an abstract function name, like 'custom_scoring_function`, probably ?

Yes