Skip to content

Joblib 0.12.2#11741

Merged
ogrisel merged 3 commits intoscikit-learn:masterfrom
ogrisel:joblib-0.12.2
Aug 3, 2018
Merged

Joblib 0.12.2#11741
ogrisel merged 3 commits intoscikit-learn:masterfrom
ogrisel:joblib-0.12.2

Conversation

@ogrisel
Copy link
Copy Markdown
Member

@ogrisel ogrisel commented Aug 3, 2018

Synchronize to the latest version of joblib.

  • includes loky: it fixes the intermittent crashes between OpenMP / Accelerate and GridSearchCV / RandomizedSearchCV;

  • makes it possible to better negotiate backends when using dask

  • better detection of the number of available CPUs and size of /dev/shm when running in containers such as docker

  • makes it possible to use LZ4 fast compression in load/dump

  • many other improvements and bug fixes.

@ogrisel ogrisel mentioned this pull request Aug 3, 2018
Closed
2 tasks
@ogrisel
Copy link
Copy Markdown
Member Author

ogrisel commented Aug 3, 2018

Ok this is good to go. Merging.

@ogrisel ogrisel merged commit 40e6c43 into scikit-learn:master Aug 3, 2018
@ogrisel ogrisel deleted the joblib-0.12.2 branch August 3, 2018 10:34
@qinhanmin2014
Copy link
Copy Markdown
Member

qinhanmin2014 commented Aug 3, 2018

@ogrisel
(1) Circle CI is failing

Unexpected failing examples:
/home/circleci/project/examples/cluster/plot_feature_agglomeration_vs_univariate_selection.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/circleci/project/examples/cluster/plot_feature_agglomeration_vs_univariate_selection.py", line 75, in <module>
    clf.fit(X, y)  # set the best parameters
  File "/home/circleci/project/sklearn/model_selection/_search.py", line 663, in fit
    cv.split(X, y, groups)))
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 981, in __call__
    if self.dispatch_one_batch(iterator):
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 818, in dispatch_one_batch
    self._pickle_cache)
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 253, in __init__
    self.items = list(iterator_slice)
  File "/home/circleci/project/sklearn/model_selection/_search.py", line 662, in <genexpr>
    for parameters, (train, test) in product(candidate_params,
  File "/home/circleci/project/sklearn/base.py", line 62, in clone
    new_object_params[name] = clone(param, safe=False)
  File "/home/circleci/project/sklearn/base.py", line 50, in clone
    return estimator_type([clone(e, safe=safe) for e in estimator])
  File "/home/circleci/project/sklearn/base.py", line 50, in <listcomp>
    return estimator_type([clone(e, safe=safe) for e in estimator])
  File "/home/circleci/project/sklearn/base.py", line 50, in clone
    return estimator_type([clone(e, safe=safe) for e in estimator])
  File "/home/circleci/project/sklearn/base.py", line 50, in <listcomp>
    return estimator_type([clone(e, safe=safe) for e in estimator])
  File "/home/circleci/project/sklearn/base.py", line 62, in clone
    new_object_params[name] = clone(param, safe=False)
  File "/home/circleci/project/sklearn/base.py", line 53, in clone
    return copy.deepcopy(estimator)
  File "/home/circleci/miniconda/envs/testenv/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/circleci/miniconda/envs/testenv/lib/python3.6/copy.py", line 274, in _reconstruct
    y = func(*args)
  File "/home/circleci/project/sklearn/externals/joblib/memory.py", line 830, in __init__
    location, cachedir))
ValueError: You set both "location='/tmp/tmp1v2mgts1' and "cachedir=False". 'cachedir' has been deprecated in version 0.12 and will be removed in version 0.14.
Please only set "location='/tmp/tmp1v2mgts1'"


/home/circleci/project/examples/compose/plot_compare_reduction.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/circleci/project/examples/compose/plot_compare_reduction.py", line 119, in <module>
    grid.fit(digits.data, digits.target)
  File "/home/circleci/project/sklearn/model_selection/_search.py", line 663, in fit
    cv.split(X, y, groups)))
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 981, in __call__
    if self.dispatch_one_batch(iterator):
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 818, in dispatch_one_batch
    self._pickle_cache)
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 253, in __init__
    self.items = list(iterator_slice)
  File "/home/circleci/project/sklearn/model_selection/_search.py", line 662, in <genexpr>
    for parameters, (train, test) in product(candidate_params,
  File "/home/circleci/project/sklearn/base.py", line 62, in clone
    new_object_params[name] = clone(param, safe=False)
  File "/home/circleci/project/sklearn/base.py", line 53, in clone
    return copy.deepcopy(estimator)
  File "/home/circleci/miniconda/envs/testenv/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/circleci/miniconda/envs/testenv/lib/python3.6/copy.py", line 274, in _reconstruct
    y = func(*args)
  File "/home/circleci/project/sklearn/externals/joblib/memory.py", line 830, in __init__
    location, cachedir))
ValueError: You set both "location='/tmp/tmpa408j1cz' and "cachedir=False". 'cachedir' has been deprecated in version 0.12 and will be removed in version 0.14.
Please only set "location='/tmp/tmpa408j1cz'"

(2) We get many extra warnings in the examples
e.g., UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = None.
(3) In the doc, we still have things like n_jobs : int, optional (default=1)
(4) What'e the difference between n_jobs=1 and n_jobs=None? Maybe add some explanations?

@js3711
Copy link
Copy Markdown

js3711 commented Aug 3, 2018

Thanks @ogrisel. Is there way to request a scikit-learn release candidate version through conda-forge (when it is ready)?

I'm excited about how dask will make use of these changes and I'd like to get a build for multiple platforms.

@jakirkham
Copy link
Copy Markdown
Contributor

jakirkham commented Aug 3, 2018

We have built RCs for scikit-learn in the past. If scikit-learn plans to make an RC, we'd be happy to work with them to get it released. RCs for various packages are available via conda-forge/label/rc.

@jnothman
Copy link
Copy Markdown
Member

jnothman commented Aug 5, 2018 via email

@rth
Copy link
Copy Markdown
Member

rth commented Aug 13, 2018

Discussion about the reported issues continued in #11771

@qinhanmin2014 qinhanmin2014 mentioned this pull request Aug 13, 2018
Closed
@rth rth mentioned this pull request Aug 13, 2018
Merged
@qinhanmin2014 qinhanmin2014 mentioned this pull request Aug 15, 2018
Merged
@TomDLT TomDLT mentioned this pull request Oct 4, 2018
Closed
@jakirkham jakirkham mentioned this pull request Nov 3, 2018
Merged
This was referenced May 24, 2019
Closed
Merged
@jakirkham jakirkham mentioned this pull request Sep 9, 2019
Merged
@jrgriffiniii jrgriffiniii mentioned this pull request Jan 26, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ogrisel @qinhanmin2014 @js3711 @jakirkham @jnothman @rth