joblib 0.13.0 by ogrisel · Pull Request #12531 · scikit-learn/scikit-learn

ogrisel · 2018-11-06T16:26:17Z

This updates joblib to 0.13.0 that now uses cloudpickle by default deterministically.
This should fix the following issues:

Fixes Serialization error when using parallelism in cross_val_score with GridSearchCV and a custom estimator #12413 (pickling error on custom estimator)
Fixes Error thrown when calling fit on RFECV of a pipeline with n_jobs=-1 in version 0.20.0 #12250 (idem)
Fixes Vendored joblib includes python 3-specific features that break python 2 #12523 (BrokenPipeError undefined on Python 2)
Fixes imp module is deprecated in favour of importlib #12474 and Cloudpickle.py, the imp module is deprecated in favor of importlib #12434 (deprecation warning for imp)

amueller · 2018-11-06T17:04:21Z

sklearn/externals/joblib/__init__.py

+    **Download:**            http://pypi.python.org/pypi/joblib#downloads

-    **Source code:**         https://github.com/joblib/joblib
+    **Source code:**         http://github.com/joblib/joblib


That's weird. There is no good reason.

amueller · 2018-11-06T17:04:57Z

So... are we going to review this? How?

amueller · 2018-11-06T17:06:50Z

sklearn/externals/joblib/pool.py

    # customize the dispatch table without side effects in Python 2.7
    # to 3.2. For Python 3.3+ leverage the new dispatch_table
-    # feature from https://bugs.python.org/issue14166 that makes it possible
+    # feature from http://bugs.python.org/issue14166 that makes it possible


all the https->http changes are a bit confusing?

ohhh did we change them with a regex in scikit-learn?

The HTTPS switch was not contributed upstream to joblib, only in scikit-learn.

Fixed upstream: joblib/joblib#805 (for the next release).

ogrisel · 2018-11-06T17:21:21Z

So... are we going to review this? How?

I would like travis and appveyor to review this :)

I also checked that the minimal reproduction cases from #12413 and #12250 now pass.

amueller · 2018-11-06T18:08:08Z

We're not running loky & joblib tests which is why codecov is failing, right?
Can we disable it on these files? And given that everything else is green, I'd say lgtm

jnothman · 2018-11-06T20:16:09Z

So... are we going to review this? How?

I think we have to review this by trusting the joblib team + CI... and then get feedback.

I've updated the PR description to auto-close four issues. Happy to merge and give it a go...

ogrisel · 2018-11-07T08:02:36Z

Thanks, I merged.

* upstream/master: joblib 0.13.0 (scikit-learn#12531) DOC tweak KMeans regarding cluster_centers_ convergence (scikit-learn#12537) DOC (0.21) Make sure plot_tree docs are generated and fix link in whatsnew (scikit-learn#12533) ALL Add HashingVectorizer to __all__ (scikit-learn#12534) BLD we should ensure continued support for joblib 0.11 (scikit-learn#12350) fix typo in whatsnew Fix dead link to numpydoc (scikit-learn#12532) [MRG] Fix segfault in AgglomerativeClustering with read-only mmaps (scikit-learn#12485) MNT (0.21) OPTiCS change the default `algorithm` to `auto` (scikit-learn#12529) FIX SkLearn `.score()` method generating error with Dask DataFrames (scikit-learn#12462) MNT KBinsDiscretizer.transform should not mutate _encoder (scikit-learn#12514)

redur · 2018-11-08T13:07:59Z

So... are we going to review this? How?

I would like travis and appveyor to review this :)

I also checked that the minimal reproduction cases from #12413 and #12250 now pass.

Hey all,

thanks for the quick fix! Unfortunately, the issue is still there for me and the reproduction case #12413 doesnt work on my machine. I also get the "deprecated in favour of importlib" warning (#12474).
I am working on Windows with:

      conda version : 4.5.11
conda-build version : 2.0.2
     python version : 3.6.6

python environment:

channels:
- defaults
- conda-forge
dependencies:
- python=3.6.6
- numpy=1.14.5
- scikit-learn=0.20.0
- joblib=0.13.0

Which seems to be installed properly.

Do you know this issue?

jnothman · 2018-11-08T23:08:16Z

It sounds like you haven't updated scikit-learn to master. Have you? We hope to release version 0.20.1 very soon

…ybutton * upstream/master: FIX YeoJohnson transform lambda bounds (scikit-learn#12522) [MRG] Additional Warnings in case OpenML auto-detected a problem with dataset (scikit-learn#12541) ENH Prefer threads for IsolationForest (scikit-learn#12543) joblib 0.13.0 (scikit-learn#12531) DOC tweak KMeans regarding cluster_centers_ convergence (scikit-learn#12537) DOC (0.21) Make sure plot_tree docs are generated and fix link in whatsnew (scikit-learn#12533) ALL Add HashingVectorizer to __all__ (scikit-learn#12534) BLD we should ensure continued support for joblib 0.11 (scikit-learn#12350) fix typo in whatsnew Fix dead link to numpydoc (scikit-learn#12532) [MRG] Fix segfault in AgglomerativeClustering with read-only mmaps (scikit-learn#12485) MNT (0.21) OPTiCS change the default `algorithm` to `auto` (scikit-learn#12529) FIX SkLearn `.score()` method generating error with Dask DataFrames (scikit-learn#12462) MNT KBinsDiscretizer.transform should not mutate _encoder (scikit-learn#12514)

redur · 2018-11-09T07:59:44Z

Ah I wasnt aware that there have been changes to scikit too. I thought it was just a joblib issue fixed with 0.13.0.

Thanks!

ogrisel · 2018-11-09T08:53:57Z

For now scikit-learn still includes its own copy of joblib in sklearn/externals/joblib. This might change in the future though :)

This reverts commit e8ddffb.

joblib 0.13.0

58457b4

ogrisel added this to the 0.20.1 milestone Nov 6, 2018

amueller reviewed Nov 6, 2018

View reviewed changes

ogrisel merged commit c81e255 into scikit-learn:master Nov 7, 2018

ogrisel deleted the joblib-0.13.0 branch November 7, 2018 08:02

thoo pushed a commit to thoo/scikit-learn that referenced this pull request Nov 14, 2018

joblib 0.13.0 (scikit-learn#12531)

25dc06f

thoo pushed a commit to thoo/scikit-learn that referenced this pull request Nov 14, 2018

joblib 0.13.0 (scikit-learn#12531)

a68e7ca

jnothman pushed a commit to jnothman/scikit-learn that referenced this pull request Nov 14, 2018

joblib 0.13.0 (scikit-learn#12531)

cb2db95

jnothman pushed a commit to jnothman/scikit-learn that referenced this pull request Nov 14, 2018

joblib 0.13.0 (scikit-learn#12531)

ffdc5c9

xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019

joblib 0.13.0 (scikit-learn#12531)

e8ddffb

xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019

Revert "joblib 0.13.0 (scikit-learn#12531)"

a48abfb

This reverts commit e8ddffb.

xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019

Revert "joblib 0.13.0 (scikit-learn#12531)"

34e4bc0

This reverts commit e8ddffb.

koenvandevelde pushed a commit to koenvandevelde/scikit-learn that referenced this pull request Jul 12, 2019

joblib 0.13.0 (scikit-learn#12531)

9099772

Uh oh!

Conversation

ogrisel commented Nov 6, 2018 • edited by jnothman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amueller Nov 6, 2018

Choose a reason for hiding this comment

Uh oh!

ogrisel Nov 6, 2018

Choose a reason for hiding this comment

Uh oh!

amueller commented Nov 6, 2018

Uh oh!

amueller Nov 6, 2018

Choose a reason for hiding this comment

Uh oh!

amueller Nov 6, 2018

Choose a reason for hiding this comment

Uh oh!

ogrisel Nov 6, 2018

Choose a reason for hiding this comment

Uh oh!

ogrisel Nov 6, 2018

Choose a reason for hiding this comment

Uh oh!

ogrisel commented Nov 6, 2018

Uh oh!

amueller commented Nov 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jnothman commented Nov 6, 2018

Uh oh!

ogrisel commented Nov 7, 2018

Uh oh!

redur commented Nov 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jnothman commented Nov 8, 2018 via email

Uh oh!

redur commented Nov 9, 2018

Uh oh!

ogrisel commented Nov 9, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ogrisel commented Nov 6, 2018 •

edited by jnothman

Loading

amueller commented Nov 6, 2018 •

edited

Loading

redur commented Nov 8, 2018 •

edited

Loading